Skip to content

fix(batch): report short-circuited FIFO records that don't match the model - #8663

Merged
leandrodamascena merged 2 commits into
aws-powertools:developfrom
soujasK:fix/fifo-short-circuit-model-error
Nov 10, 2026
Merged

leandrodamascena merged 2 commits into
aws-powertools:developfrom
soujasK:fix/fifo-short-circuit-model-error

Conversation

@soujasK

@soujasK soujasK commented Oct 25, 2026

Copy link
Copy Markdown
Contributor

Issue number: closes #8661

Summary

Changes

Once a record fails, SqsFifoPartialProcessor short-circuits the remaining records (or the rest of the failed group with skip_group_on_error=True) and converts them with the customer's model before passing them to failure_handler. That conversion was outside any try, so a record that didn't match the model raised out of process() and failed the whole invocation.

Short-circuited records never reach the record handler, so if the model conversion fails they're now registered as event source data classes, like other poison pills. Records that do match the model are still registered as model instances, so fail_messages doesn't change for them.

Tests: a batch with a successful record, a failing record, a record that doesn't match the model in the same group, and a successful record in another group, parametrized over skip_group_on_error. Both cases raised ValidationError before the change and pass after it. tests/functional/batch passes on Linux (Python 3.12), along with ruff, mypy, ty, bandit and xenon.

User experience

Before: a malformed record after a failure fails the whole invocation, so every record is retried, including already processed ones (and, with skip_group_on_error=True, the successful records of other message groups).

After: the malformed record is reported in batchItemFailures along with the other short-circuited records.


By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@soujasK
soujasK requested a review from a team as a code owner October 25, 2026 18:05
@soujasK
soujasK requested a review from svozza October 25, 2026 18:05
@boring-cyborg boring-cyborg Bot added the tests label Oct 25, 2026
@powertools-for-aws-oss-automation powertools-for-aws-oss-automation Bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Oct 25, 2026
@leandrodamascena leandrodamascena added this to the 3.40.0 milestone Oct 25, 2026
@soujasK
soujasK force-pushed the fix/fifo-short-circuit-model-error branch from 92dc822 to 1545ba0 Compare October 26, 2026 10:41
@soujasK soujasK closed this Oct 31, 2026
@leandrodamascena

Copy link
Copy Markdown
Contributor

Hi @soujasK, I saw you closed this one. No problem, I just want to understand what you'd prefer. The bug is real and triaged for 3.40.0, and I already worked on it on my side. Do you want to reopen so I review your PR, or should I open a new one myself? Either way is fine, just let me know.

@soujasK soujasK reopened this Oct 31, 2026
@soujasK

soujasK commented Oct 31, 2026

Copy link
Copy Markdown
Contributor Author

Reopening this. I closed it by mistake. I need a bit more time to test and validate it properly, so I'll update here once that's done.

@soujasK

soujasK commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

please review mine

@leandrodamascena leandrodamascena left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @soujasK, and sorry for the wait. I ran this on top of today's develop in a clean environment: the batch suite, the required-dependencies session, ruff, mypy and xenon, all green. The change is exactly what the issue needed: a record that does not match the model after a short circuit becomes one more entry in batchItemFailures instead of failing the invocation. This goes out in 3.40.0, no rebase needed, credit in the release notes.

@sonarqubecloud

Copy link
Copy Markdown

@codecov

codecov Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.18%. Comparing base (93159ce) to head (43685a5).
⚠️ Report is 1 commits behind head on develop.

Additional details and impacted files
@@           Coverage Diff            @@
##           develop    #8663   +/-   ##
========================================
  Coverage    97.18%   97.18%           
========================================
  Files          323      323           
  Lines        16542    16546    +4     
  Branches      1495     1495           
========================================
+ Hits         16076    16080    +4     
  Misses         326      326           
  Partials       140      140           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@leandrodamascena
leandrodamascena merged commit bf90748 into aws-powertools:develop Nov 10, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/M Denotes a PR that changes 30-99 lines, ignoring generated files. tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: SqsFifoPartialProcessor fails the whole batch when a short-circuited record doesn't match the model

2 participants